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The School District of Philadelphia (SDP) is a partner in the Free Library of Philadelphia’s Read by 4th, a citywide 
initiative that aims to substantially increase the percentage of students who can read on grade level when they 
enter grade 4. To measure the percentage of entering kindergarteners on-track to meet this goal, SDP 
partnered with the Regional Educational Laboratory Mid-Atlantic to identify an indicator based on the 
Pennsylvania Kindergarten Entry Inventory (KEI). Students with higher scores on the KEI in 2014/15 had higher 
English language arts scores on Pennsylvania’s grade 3 state assessment in 2017/18. The study established the 
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Engagement Competencies) to predict whether students would be proficient in English Language Arts in grade 
3. The percentage of students above this threshold matched the actual 37 percent proficiency rate of the study 
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the proficiency status of 53 percent of the students who were proficient on the grade 3 state assessment and 
73 percent of those who were not proficient. Students with higher scores on SDP’s annual spring reading 
assessments in kindergarten through grade 3 also had higher KEI scores and English language arts scores on 
the grade 3 state assessment. 


In the 2017/18 school year, nearly two-thirds of students in the School District of Philadelphia (SDP) were not 
proficient in reading by the end of grade 3 (SDP, 2019). Research has shown that students who fail to read at grade 
level by the end of grade 3 have a higher risk of negative long-term outcomes, such as failing to reach proficiency 
on sixth-grade reading and math assessments, grade retention, and dropping out of high school (Connor et al., 
2011; Massachusetts Department of Elementary and Secondary Education and American Institutes for Research, 
2013; Phillips, Norris, Osmond, & Maynard, 2002; Hernandez, 2012). 


SDP is a partner in the Free Library of Philadelphia’s Read by 4th Campaign, a citywide initiative that, by 2020, 
aims to substantially increase the percentage of students who can read on grade level when they enter grade 4. 
More than 100 partner organizations help support Philadelphia’s families and their children from birth through 
grade 3 in developing early language and literacy skills through family activities, pre-kindergarten experiences, 
and school and out-of-school experiences in kindergarten through grade 3. SDP has also made early literacy a 
focus from kindergarten through grade 3, setting as one of its four strategic “anchor” goals: “100 percent of 8- 
year-olds will read on or above grade level” (City of Philadelphia, 2017). To meet this goal, SDP has strengthened 
its early literacy curriculum and assessment systems, provided a literacy coach in every elementary school, 
strengthened classroom literacy environments, and distributed one million new books across the district’s 
kindergarten through grade 3 classrooms. 


To help track progress toward the goal of proficient reading by the end of grade 3, SDP requested assistance from 
the Regional Educational Laboratory (REL) Mid-Atlantic to identify an indicator measuring the percentage of 
entering kindergarteners on-track for proficiency in English Language Arts (ELA) on the grade 3 Pennsylvania 
System of School Assessment (PSSA). The indicator can provide useful information on the progress of families and 
organizations across the city to promote children’s early development before kindergarten entry and inform the 
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city’s continued investments in early education. This indicator would be based on Pennsylvania’s Kindergarten 
Entry Inventory (KEI), an assessment that measures children’s academic, socioemotional, and physical 
competencies that has been used in the district since 2014. 


SDP also wanted to explore the accuracy of using this indicator or alternative thresholds on the KEI, along with its 
other measures of early reading skills, to predict which individual students may need support in reading. In 
addition, SDP wanted to understand the relationships between its different measures of early reading skills—the 
KEI, PSSA, and AlMSweb assessments (see box 1 for key terms). SDP currently uses the thresholds set by the 
developer on the AlMSweb to identify individual students who need additional support. Therefore, the study 
examines the accuracy of developer-selected thresholds on the AlMSweb for predicting which individual students 
are at risk of not being proficient in reading on the grade 3 PSSA, the accuracy of thresholds set on the KEI for such 
predictions, and the relationships between the different measures of early reading skills. This will inform SDP’s 
decisions about the measures used to predict which students may need support in reading and possible future 
work on other indicators of being on-track for reading proficiency. 


Box 1. Key terms 
Accuracy. The study examines four measures of accuracy (see box A1 in appendix A for equations): 


e Absolute difference between the actual proficiency rate and predicted proficiency rate based on the threshold. This 
measures how accurately the threshold predicts the percentage of entering kindergarteners in the study cohort who are 
on-track for reading proficiency. 

e Percentage of proficient students whom the threshold correctly predicted. This measures how accurately the threshold 
made predictions for individual students who are proficient. 

e Percentage of not proficient students whom the threshold correctly predicted. This measures how accurately the threshold 
made predictions for individual students who are not proficient. 

e Percentage of students whom the threshold correctly predicted. This measures how accurately the threshold made 
predictions for individual students (both those who were proficient and those who were not). 


AlMSweb Reading. The AlMSweb Reading assessment measures children’s progress in developing reading skills between 
kindergarten and grade 3. The AlMSweb includes subtests that measure early literacy skills, such as letter naming fluency 
(naming upper and lower case letter names) in kindergarten, nonsense word fluency (associating letters with sounds and 
blending letter sounds into words) in grades 1 and 2, and oral reading fluency in grade 3. 


Teachers administer the AlMSweb three times (fall, winter, and spring) during each school year, and each subtest takes about 
1 to 4 minutes. This study focuses on the spring scores as a measure of students’ reading skills by the end of each grade. This 
study was not able to use the AlMSweb as an indicator for the percentage of entering kindergarteners on-track for reading 
proficiency because the district did not require the fall AIMSweb until 2015. 


AlMSweb scores divide students into three tiers (1, 2, and 3) that are consistent with Response to Intervention, a widely used 
approach for identifying and supporting struggling students. In Tier 1, students are considered “at target” and receive typical 
instruction and periodic screening. In Tier 2, struggling students receive strategic interventions and more frequent progress 
monitoring. In Tier 3, students receive intensive intervention. 


Kindergarten Entry Inventory (KEI). Since 2014, SDP has required schools to administer the KEI. The KEI measures children’s 
academic, socioemotional, and physical competencies at kindergarten entry using 30 indicators. A previous validation study 
identified two distinct dimensions on the KEI with evidence of validity (Howard et al., 2017). This study focuses on these two 
dimensions, as well as an index of the two dimensions constructed by the study team: 


e Emerging Academic Competencies, which includes 12 indicators that measure early literacy and mathematics skills, such 
as letter recognition and counting. 

e Learning Engagement Competencies, which includes nine indicators that measure behavioral and motor skills, such as task 
persistence, emotion regulation, and fine and gross motor skills. 
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e Index of Emerging Academic Competencies and Learning Engagement Competencies, which is a weighted average of these 
dimensions, with greater weight given to Emerging Academic Competencies, which has a stronger relationship with ELA 
scores on the grade 3 PSSA than Learning Engagement Competencies (equation A2 in appendix A). 


Teachers administer the KEI by observing children in a variety of classroom activities during the first 45 days of instruction. 
Teachers give children a rating on each indicator using four rating levels: (1) Not yet evident, (2) Emerging, (3) Evident, and 
(4) Exceeds. For each dimension, the score is the average rating across the indicators and ranges from 1 to 4. The study team 
put the index on a 1 to 10 scale, in consultation with SDP, to make this scale more distinct from those of the individual 
dimensions. On the index, 10 indicates students scored perfectly (were rated as 4, Exceeds) on all items on the two KEI 
dimensions used. 


Pennsylvania System of School Assessment (PSSA) in English language arts. This state standardized assessment measures 
students’ reading skills, including decoding and comprehension, in the spring of grade 3. On the PSSA, performance levels can 
be classified as proficient (proficient or advanced) or not proficient (below basic or basic). More details about the indicators 
in each KEI dimension and the subtests included in the AlMSweb is in appendix A. 


Research questions 


To identify an indicator of being on-track to read proficiently by the end of grade 3, this study addressed several 
research questions for the cohort of students who entered kindergarten in SDP in the 2014/15 school year. This 
cohort was the first to be administered the current version of the KEI assessment in fall 2014 and the only cohort 
for whom grade 3 PSSA scores were available when the study began: 


Primary research questions: 

1. What is the relationship between KEI scores and scores on the grade 3 PSSA in ELA? 

2. What threshold score on the KEI most accurately predicts reading proficiency on the grade 3 PSSA in ELA for 
the cohort? What threshold score on the KEI has a correct prediction rate of at least 90 percent for the 
individual students who were not proficient in ELA on the grade 3 PSSA? 


Exploratory research question: 


3. How do AlMSweb Reading scores in the spring of kindergarten through grade 3 relate to children’s scores on 
the KEI and grade 3 PSSA in ELA? 


The primary research questions inform the development of an indicator of progress towards grade-level reading 
based on KEI scores. The relationship between KEI scores and the scores on the grade 3 PSSA provides information 
about the potential for using a KEI threshold for predicting reading proficiency in grade 3. The most accurate KEI 
threshold score for predicting the percentage of students that are on track for grade-level reading—SDP’s primary 
use for the indicator—is identified. 


Should SDP decide to use the KEI, in combination with other measures, to predict which individual children are at 
risk of not meeting grade level reading expectations, a threshold that accurately predicted more of the students 
who are not proficient might be more appropriate. Studies that aim to identify students at risk of not being 
proficient in reading often try to set thresholds on screening instruments that correctly identify 90 percent of 
those who are not proficient (Compton, Fuchs, Fuchs, & Bryant, 2010; Jenkins, Hudson, & Johnson, 2007). Early 
identification could allow these children to receive further assessment and potentially additional services to 
support reading growth. Yet, such a threshold should also minimize the risk of spending limited resources 
supporting students who may not need extensive supports. Therefore, the accuracy of alternative thresholds to 
predict which individual students are most likely to need support in reading is explored. 


The final research question explores the relationship between the different measures of early reading skills used 
in SDP—the KEI, AlMSweb, and PSSA. Because SDP was not using AlIMSweb scores across the district in fall 2014, 
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the study focused on spring AlMSweb scores between kindergarten and grade 3. The relationships between KEI 
scores, AlMSweb scores from the spring of kindergarten through grade 3, and grade 3 PSSA scores in ELA provide 
information about how students’ reading proficiency changes across grades and how it is classified by different 
measures. For example, the AlIMSweb could be more accurate at predicting grade 3 reading proficiency because 
it measures more specific reading skills than the validated KEI dimensions, which measure literacy, math, and 
behavioral skills. The AlIMSweb may also help identify the specific reading skills that reading interventions should 
target. Therefore, information about the relationship between AlMSweb and the PSSA can also inform SDP’s 
decisions about measures to identify students at risk of not being proficient in reading. However, these analyses 
were exploratory because SDP’s primary goal was to develop an indicator based on the KEI that could track 
entering kindergarteners’ progress towards reading proficiency. 


The data sources, sample, and methods used to answer these research questions are described in box 2. 


Box 2. Data sources, sample, and methods 


Data sources. This study used administrative student records from the School District of Philadelphia (SDP) that included 
information about students’ school enrollments and demographic characteristics, Kindergarten Entry Inventory (KEI) scores 
on the two dimensions, grade 3 Pennsylvania System of School Assessment (PSSA) in English Language Arts (ELA) scores, and 
AlMSweb Reading scores for each spring semester from kindergarten to grade 3. 


Sample. The study sample was 13,335 kindergartners who entered SDP by December of the 2014/15 school year. The sample 
did not include students in charter or private schools in Philadelphia. Of these 13,335 students, 58 to 68 percent had scores 
on one or both KEI dimensions. Students with both KEI dimension scores were more likely to be White than all kindergarteners 
(table A7 in appendix A). In addition, of students with KEI scores, 65 percent also had ELA scores on the grade 3 PSSA (table 
A4 in appendix A). 


For research questions 1 and 2, the sample was restricted to a random sample of 70 percent of the students who had scores 
on both KEI dimensions in 2014 and took the grade 3 PSSA on-time in 2018 (3,521 students; see the methods below for 
explanation of why analyses used a random sample of 70 percent of students). An analysis for research question 2 also used 
the remaining 30 percent random sample of students (1,508 students). The samples for research question 3 were restricted 
to a 70 percent random sample and included enrolled kindergarteners who completed the spring AlMSweb and either the 
KEI or PSSA in each of the years that the 2014/15 kindergarteners were expected to complete them if they remained on grade 
level (3,689 to 5,446 students). 


Methods. Following the typical approach for predictive analyses, the study team split the sample randomly into two parts, 
using a method that ensured that each part contained the same percentages of students who were and were not proficient 
on the PSSA. The study team conducted most analyses on 70 percent of the sample to select the threshold (described as the 
sample used to set the threshold) and then checked the accuracy of the predictions using the remaining 30 percent. 


The study team used descriptive statistics and regression analyses to analyze the relationships between KEI and PSSA scores 
(research question 1). To assess the relationship between each KEI dimension score and students’ PSSA score, the study team 
used pairwise correlations. Positive correlations were classified as follows: weak (0.1-0.39), moderate (0.4—0.69), strong (0.7— 
0.99), and perfect (1; Dancey & Reidy, 2007). The study team used logistic regression analyses to predict the likelihood that 
a student was proficient on the grade 3 PSSA based on their scores on the KEI dimensions (both separately and combined). 
To make the sample used in the analysis (the sample of kindergarteners with both KEI and PSSA scores) representative of the 
sample to whom the threshold will be applied (the sample of kindergarteners with KEI scores), the study team used weights 
for the analyses for research questions 1 and 2. These weights adjust for the fact that some students with KEI scores were 
missing PSSA scores, and thus, are excluded from the predictions. Students with PSSA scores had higher KEI scores on both 
dimensions, were more likely to be Hispanic/Latino, and were less likely to receive special education services than students 
without PSSA scores (table A8 in appendix A). The analysis adjusts for the exclusion of these students by giving greater weight 
to students with similar KEI scores and characteristics who did have PSSA scores (findings using alternative weights are shown 
in appendix B). 
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To select a threshold and assess its accuracy (research question 2), the study team and SDP first considered three potential 
indicators on which the threshold could be set: Emerging Academic Competencies dimension only, Learning Engagement 
Competencies dimension only, or an index of both Emerging Academic and Learning Engagement Competencies (see 
appendix A for detail on the construction of the index). To provide information about deciding between these three potential 
indicators, the study team calculated two statistics that provide information about the accuracy of thresholds for predicting 
individual students’ proficiency: the percentages of proficient students and students who were not proficient whom the 
threshold correctly predicted. Over the course of the study, the study team and SDP staff discussed these different indicators, 
and SDP decided to use an index based on both dimensions as the indicator of whether students were on-track for reading 
proficiently in grade 3. The study team then identified the threshold for this index that had the smallest absolute difference 
between the actual and predicted proficient rates and rounded to the nearest whole number to more easily communicate 
the threshold. 


To help shed light on the potential usefulness of the overall threshold for predicting reading proficiency for future cohorts of 
students, the accuracy of the threshold was also assessed for the remaining 30 percent sample of students (who had not 
been used to set the threshold), based on these three statistics and the percentage of all students whom the threshold 
correctly predicted. As an exploratory analysis, the study team also used this method to explore the accuracy of alternative 
thresholds on the KEI index that made correct predictions for at least 90 percent of individual students who were at risk of 
not being proficient in ELA on the grade 3 PSSA. 


Descriptive statistics were used to explore the relationships between AlMSweb and KEI and PSSA scores (research question 
3). As with research question 1, pairwise correlations were calculated between students’ scores on each assessment. Weights 
were not used in the analyses that included the AlMSweb scores because the analyses were exploratory and not intended to 
set an indicator that could be applied to a representative population of students (such as all students with AlIMSweb scores 
in a particular grade). Measures of accuracy (correctly predicting students who were and were not proficient on the grade 3 
PSSA in ELA) were calculated for the developer-selected AlMSweb thresholds, using the equations shown in box A1 in 
appendix A. 


More information about the study data sources, sample, and methods is in appendix A. 


Findings 
This section presents findings to address the three study research questions. Additional findings are available in 
appendix B. 


Students with higher Kindergarten Entry Inventory scores had higher scores on the grade 3 
Pennsylvania System of School Assessment in English language arts 


Students’ KEI scores were weakly to moderately related to ELA scores on the grade 3 PSSA. Emerging Academic 
Competencies had a correlation of 0.42 with ELA scores on the grade 3 PSSA, and Learning Engagement 
Competencies had a correlation of 0.32 with the same grade 3 test scores (table 1). The stronger correlation 
between Emerging Academic Competencies and grade 3 PSSA in ELA scores, relative to that for Learning 
Engagement Competencies, may reflect the fact that the Emerging Academic Competencies dimension includes 
measures of early reading skills (as well as math skills). In contrast, the Learning Engagement Competencies 
dimension includes measures of behavioral and motor skills that are less likely to be strongly correlated with 
grade 3 reading skills. 
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Table 1. Both Kindergarten Entry Inventory dimensions had weak to moderate correlations with grade 3 
Pennsylvania System of School Assessment scores 


KEI dimension Correlation with grade 3 PSSA scores 


Emerging Academic Competencies 0.42*** 
Learning Engagement Competencies 0.32*** 


KEI is Kindergarten Entry Inventory. PSSA is Pennsylvania System of School Assessment. 

* Significant at p = .05; ** significant at p = .01; *** significant at p = .001. 

Note: Analysis based on the random sample of students with both KEI and grade 3 PSSA scores that was used to set the threshold and weighted to represent 
all students with KEI scores. 3,521 students had information on both KEI competencies scores and grade 3 PSSA in ELA. 

Source: Authors’ analysis based on School District of Philadelphia 2014/15 and 2017/18 data described in appendix A. 


Students with higher KEI scores were more likely to receive a score in the proficient level on the grade 3 PSSA in 
ELA. For both Emerging Academic Competencies and Learning Engagement Competencies, PSSA proficiency 
rates ranged from less than 15 percent for students with the lowest scores on each KEI dimension (1-1.5) to 
over 60 percent for students with the highest scores on each dimension (3.6—4; figure 1). However, PSSA 
proficiency rates increased more steeply with scores on Emerging Academic Competencies (left graph) than with 
those on Learning Engagement Competencies (right graph), consistent with the higher correlation found for 
Emerging Academic Competencies. 


Figure 1. Students with higher Kindergarten Entry Inventory scores were more likely to be proficient on the 
grade 3 Pennsylvania System of School Assessment in English Language Arts 
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ELA is English language arts. KEI is Kindergarten Entry Inventory. PSSA is Pennsylvania System of School Assessment. 

Note: Analysis based on the random sample of students with both KEI and grade 3 PSSA scores that was used to set the threshold and weighted to represent 
all students with KEI scores. 3,521 students had information on both KEI competencies scores and grade 3 PSSA in ELA. 

Source: Authors’ analysis based on School District of Philadelphia 2014/15 and 2017/18 data described in appendix A. 


After accounting for Emerging Academic Competencies, Learning Engagement Competencies had little 
relationship with ELA scores on the grade 3 PSSA. When examined separately, both Emerging Academic 
Competencies and Learning Engagement Competencies had statistically significant relationships with students’ 
grade 3 PSSA proficiency in ELA (model 1 and model 2, table 2). On average, students who scored 1-point higher 
(on the 1-to-4 point scale) on Emerging Academic Competencies were 25 percent more likely to be proficient on 
the grade 3 PSSA (model 1), and students who scored 1-point higher on Learning Engagement Competencies 
were 24 percent more likely to be proficient on the grade 3 PSSA (model 2). When both dimensions were 
included (model 3), only Emerging Academic Competencies had a statistically significant relationship with grade 
3 PSSA proficiency in ELA, which may be because Learning Engagement Competencies was strongly correlated 
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with Emerging Academic Competencies (correlation = 0.73) (not shown). In addition, the model that included 
both KEI dimensions had the same ability to explain proficiency as the model that only included Emerging 
Academic Competencies (as evidenced by these models having the same Pseudo R-squared statistic) (table 2). 
This suggests that after accounting for students’ early reading and math skills (as measured by Emerging 
Academic Competencies), the behavioral and motor skills measured by Learning Engagement Competencies do 
not further explain students’ grade 3 PSSA proficiency. 


Table 2. Emerging Academic Competencies were more predictive of students’ probability of being proficient 
on the grade 3 Pennsylvania System of School Assessment than were Learning Engagement Competencies 
Relationship between KEI scores and the probability of being proficient on the grade 


3 PSSA in ELA 
Model 2: Learning 
Model 1: Emerging Engagement 
KEI dimensions Neor-Yol=¥aalcom Golan) oX-1n-y ales) Competencies W/KoYe (=e sYol dame) fant-lakielaks 
Emerging Academic Competencies 0.25*** 0.24*** 
Learning Engagement Competencies 0.24*** 0.02 
Pseudo R-squared 0.09 0.05 0.09 


ELA is English language arts. KEI is Kindergarten Entry Inventory. PSSA is Pennsylvania System of School Assessment. 

* Significant at p = .05; ** significant at p = .01; *** significant at p = .001. 

Note: Analysis based on the random sample of students with both KEI and grade 3 PSSA scores that was used to set the threshold and weighted to represent 
all students with KEI scores. 3,521 students had information on both KE] competencies scores and grade 3 PSSA in ELA. Reported values are average marginal 
effects, or the average increase in the predicted probability of PSSA proficiency associated with a 1-point increase in a KEI score. That is, in model 1, for every 
1-point increase in Emerging Academic Competencies, the probability of proficiency increases on average 25 percent. Pseudo R-squared is a measure of how 
well the model fits the data compared to a model with no predictors. 

Source: Authors’ analysis based on SDP 2014/15 and 2017/18 data described in appendix A. 


A threshold of 6 out of 10 on an index of Emerging Academic Competencies and Learning 
Engagement Competencies accurately predicted the actual grade 3 proficiency rate of students (37 
percent). 

Thresholds on Emerging Academic Competencies and the index based on both dimensions had similar rates of 
correctly predicting students’ proficiency. The threshold of 6 on the index of both dimensions correctly predicted 
the proficiency status of 66 percent of students. 


Thresholds based on the Emerging Academic Competencies score or the index that included this score more 
accurately predicted grade 3 PSSA proficiency in ELA than the Learning Engagement Competencies score alone. 
For example, the thresholds on Emerging Academic Competencies and the index of both dimensions that correctly 
predicted about 50 percent of proficient students also correctly predicted about 75 percent of students who were 
not proficient (figure 2). There is no threshold on Learning Engagement Competencies that performs better. 


When identifying a threshold, there are tradeoffs between correctly predicting the proficiency status of students 
who were proficient and those who were not (as shown in figure 2). Thresholds that are set closer to the lowest 
scores will increase correct predictions for proficient students at the cost of decreasing correct predictions for 
students who were not proficient; setting a threshold closer to the highest possible score will have the opposite 
effect. 


To decide whether to use Emerging Academic Competencies or the index of both dimensions as the indicator, SDP 
considered how much information each one included. SDP chose to set a threshold on the index of Emerging 
Academic Competencies and Learning Engagement Competencies for the indicator because it used more of the 
information available on children’s kindergarten competencies. 
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Figure 2. Thresholds based on indices that included Emerging Academic Competencies correctly predicted 
more proficient students and more students who were not proficient than those based on only Learning 
Engagement Competencies 
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ELA is English language arts. KEI is Kindergarten Entry Inventory. PSSA is Pennsylvania System of School Assessment. 

Note: Analysis based on the random sample of students with both KEI and grade 3 PSSA scores that was used to set the threshold and weighted to represent 
all students with KEI scores. 3,521 students had information on both KEI competencies scores and grade 3 PSSA in ELA. 

Figure reads: On Learning Engagement Competencies, the threshold that correctly predicted 50 percent of students who were not proficient also correctly 
predicted about 71 percent of proficient students. 

Source: Authors’ analysis based on School District of Philadelphia 2014/15 and 2017/18 data described in appendix A. 


At a threshold of 6 out of 10 on the index using both KEI dimensions, the percentage of students above the threshold 
closely matched the actual proficiency rate of students. This was true for both the sample used to select the 
threshold and the sample used to test the threshold. The selected threshold accurately predicted the percentage 
of proficient students for the sample of kindergarteners from 2014/15 that had not been used to select the 
threshold (table 3). This evidence supports the threshold’s potential accuracy when applied to new data for future 
cohorts of kindergarteners. 


The sample used to create the threshold is based on students who entered SDP in kindergarten, had KEI scores, 
made normal grade progress, and took the grade 3 PSSA. However, the predicted proficiency rates are also similar 
to the proficiency for all SDP grade 3 students in 2018 (36 percent), providing additional evidence of the validity 
of this indicator and threshold (SDP, 2019). 
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Table 3. The threshold had similar rates of accuracy in samples that were and were not used to select the 
threshold 
PNosxo) [hn Correct predictions for 
difference 
between actual 


and predicted 


Actual acre llancte| proficiency Proficient Not proficient Number 

proficiency proficiency (percentage students students All students of 
Sample (percentage) (percentage) rexey fale) (percentage) (percentage) (percentage) students 
Used to 37 37 1 53 73 66 3,521 
set the 
threshold 
Not used 37 37 0 58 76 69 1,508 
to set the 
threshold 


KEI is Kindergarten Entry Inventory. 

Note: Models were weighted to represent all students with KEI scores. 

The difference between actual and predicted proficiency may not equal the difference between the actual and predicted proficiency columns in the table 
due to rounding. 

Source: Authors’ analysis based on School District of Philadelphia 2014/15 and 2017/18 data described in appendix A. 


The threshold selected to make predictions for the cohort of kindergarten students made correct predictions for 
53 percent of proficient students and 73 percent of students who were not proficient. In the sample used to 
select the threshold, 37 percent of students were above the threshold, and thus predicted to be proficient (table 
3; areas A and C in figure 3), matching the percentage who were actually proficient (table 3; areas A and B in 
figure 3). However, this does not mean that the threshold correctly predicted the proficiency of every individual 
student (figure 3). It means that the incorrect predictions for individual students cancel each other out in the 
aggregate (that is, areas B and C in figure 3 are equal). Overall, the threshold correctly predicted the proficiency 
status of 66 percent of students, which is higher than the 50 percent expected to be correctly predicted by 
chance, but still results in some errors for individual students. This illustrates the fact that an indicator can be 
very accurate at the aggregate level even while making errors at the individual level. 


Figure 3. A threshold of 6 resulted in the percentage of students above the threshold matching the actual 
proficiency rate 
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ELA is English language arts. KEI is Kindergarten Entry Inventory. PSSA is Pennsylvania System of School Assessment. 

Note: Analysis based on the random sample of students with both KEI and grade 3 PSSA scores that was used to set the threshold and weighted to represent 
all students with KEI scores. 3,521 students had information on both KEI competencies scores and grade 3 PSSA in ELA. Data were aggregated to smooth out 
the distribution, so no data are displayed at exactly 1 or 10. 

Figure reads: At this threshold, students who were actually proficient (areas A and B) were either correctly predicted to be proficient (area A) or incorrectly 
predicted to be not proficient (area B). Students who were actually not proficient (areas D and C) were either correctly predicted to be not proficient (area 
D) or incorrectly predicted to be proficient (area C). 

Source: Authors’ analysis based on School District of Philadelphia 2014/15 and 2017/18 data described in appendix A. 


A threshold of 7 on the index of both dimensions made correct predictions for 91 percent of students who were not 
proficient on the PSSA and only 29 percent of students who were proficient. Compared with the threshold of 6, the 
higher threshold of 7 correctly predicted more of the students who were not proficient and fewer of the proficient 
students (table 4; figure B1 in appendix B; see table B4 in appendix B for statistics for additional thresholds). 


Table 4. A threshold of 7 on the index of both Kindergarten Entry Inventory dimensions correctly predicted 91 
percent of students who were not proficient and 29 percent of proficient students 


Noaclutia aittaranee Correct predictions for 


al alacsvare)(eMeamdar-mialel =). Predicted between actual and Proficient Not proficient 

using both KE] proficiency predicted proficiency students students All students 
olfaatesielaks (percentage) (percentage points) (percentage) (percentage) (percentage) 
6.00 37 1 53 73 66 

7.00 17 21 29 91 68 


Note: Analysis based on the random sample of students with both KEI and grade 3 PSSA scores that was used to set the threshold and weighted to represent 
all students with KEI scores. 
Source: Authors’ analysis based on SDP 2014/15 and 2017/18 data described in appendix A. 


Students with higher AlMSweb scores in kindergarten through grade 3 had generally higher 
Kindergarten Entry Inventory scores and grade 3 ELA scores on the Pennsylvania System of School 
Assessment 

Each measure of students’ reading skills from kindergarten through grade 3 were positively correlated with one 
another. Findings suggest that thresholds on the KEI could also be used to predict whether student cohorts are 


| REL Mid-Atlantic | Using kindergarten entry assessments 10 


on-track for meeting the grade-level target on the grade 3 AlMSweb.! The results also suggest that AIMSweb 
scores could be used, along with additional measures, to predict which students are at risk for not being proficient 
in reading by grade 3. However, the KEI score threshold resulted in more accurate predictions of the overall 
percentage of students proficient in reading by grade 3 than the annual AlMSweb grade-level targets. 


Students’ KEI scores on Emerging Academic Competencies generally had moderate positive correlations with their 
AlMSweb scores from spring of kindergarten through grade 3. Emerging Academic Competencies scores had 
generally moderate associations with AlMSweb scores (0.39 to 0.51), as opposed to the weak associations 
between AlMSweb and Learning Engagement Competencies scores (0.29 to 0.39; table 5). This likely reflects 
Emerging Academic Competencies being a better measure of students’ early reading skills than Learning 
Engagement Competencies. 


The KEI scores on both Learning Engagement Competencies and Emerging Academic Competencies had the 
strongest associations with spring kindergarten AlMSweb scores (0.39 and 0.51 respectively). The associations 
between KEI and AlMSweb scores generally decreased as students moved from kindergarten through grade 3. 
This is consistent with students’ early reading proficiency changing over time, and past research that shows 
stronger correlations between measures at closer points in time (Duncan et al., 2007). Associations between the 
KEI and the AlMSweb in the spring of grade 3 were identical to those between the KEI and ELA scores on the PSSA 
in grade 3. This suggests that the KEI could also be used to predict whether student cohorts are on-track for 
meeting their grade-level target on the spring AlMSweb assessments in grade 3 or earlier. 


Table 5. Kindergarten through grade 3 AlMSweb scores had weak to moderate correlations with Emerging 
Academic Competencies and Learning Engagement Competencies 


eee Spring grade 1 Springgrade2 Spring grade 3 PSSA ELA 
KEI (fall of kindergarten) AlMSweb AlMSweb AlMSweb AlMSweb grade 3 
Correlations 
Emerging Academic Competencies 0.51*** 0.39*** 0.46*** 0.42*** 0.42*** 
Learning Engagement 0.39*** 0.29*** 0.33*** 0.32*** 0.32*** 
Competencies 
Number of students 
Emerging Academic Competencies 4,930 3,689 3,948 3,707 3,697 
Learning Engagement 5,446 4,200 4,460 4,177 4,169 


Competencies 


KEI is Kindergarten Entry Inventory. PSSA is Pennsylvania System of School Assessment. 

* Significant at p = .05; ** significant at p = .01; *** significant at p = .001. 

Note: Analysis was based on the random sample of students used to set the threshold and was restricted to students who completed the KEI and either the 
AlMSweb or PSSA. Correlations for AlMSweb scores were not weighted. 

Source: Authors’ analysis based on SDP 2014-2018 data described in appendix A. 


Students’ AlMSweb scores from kindergarten through grade 3 were moderately to strongly related to their ELA 
scores on the grade 3 PSSA. The strength of the associations between AlMSweb scores and grade 3 PSSA scores 
generally increased from kindergarten to grade 3 (0.50 to 0.73; table 6). Correlations were stronger between the 
AlMSweb and PSSA than between both KEI dimensions and the PSSA. This could be because the AlMSweb was 
measured closer in time to the PSSA or could indicate that the skills measured by the AlMSweb are a stronger 
predictor of the skills measured by the PSSA. These findings suggest that the AlMSweb in grades K through 3 could 
be used to develop an indicator of students’ likelihood of being proficient in ELA on the grade 3 PSSA. 


1 For more information about the relationship between the KEI and AlMSweb tiers, see figures B1-B4 in appendix B. 
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Table 6. Kindergarten through grade 3 AlMSweb scores were moderately to strongly correlated with grade 3 
Pennsylvania System of School Assessment English Language Arts scores 


Grade 3 PSSA ELA 
Assessment Correlation Number of students 


KEI 

Emerging Academic Competencies (fall kindergarten) 0.42*** 3,697 
Learning Engagement Competencies (fall kindergarten) 0.32*** 4,169 
AlMSweb 

Spring kindergarten AlIMSweb 0.50*** 4,696 
Spring grade 1 AlMSweb Q.53"** 4,174 
Spring grade 2 AlIMSweb 0.73"** 5,027 
Spring grade 3 AlIMSweb O72"** 5,266 


KEI is Kindergarten Entry Inventory. PSSA is Pennsylvania System of School Assessment. 

* Significant at p = .05; ** significant at p = .01; *** significant at p = .001. 

Note: Analysis was based on the random sample of students used to set the threshold and is restricted to students who completed the PSSA and either the 
KEI or AlMSweb. Correlations for AlIMSweb and PSSA scores were not weighted. For comparison, unweighted correlations for the KEI and PSSA are shown. 
Source: Authors’ analysis based on School District of Philadelphia 2014—2018 data described in appendix A. 


Students in higher performing AlMSweb tiers in the spring of each year were more likely to be proficient on the 
grade 3 PSSA in ELA. SDP uses cutoffs on the AlMSweb to identify individual students for support in reading. More 
than half of the students who meet the AlMSweb’s grade-level target (Tier 1) in spring of kindergarten were 
proficient in reading on the PSSA by grade 3 (figure 4). In contrast, less than 25 percent of students identified for 
intensive support (Tier 2) and less than 10 percent of those identified for the most intensive support (Tier 3) in 
kindergarten were proficient in reading on the PSSA by grade 3 (table B5 in appendix B). These classifications 
become increasingly accurate as assessments become closer to the grade 3 PSSA. 


Figure 4. Students in higher performing AlMSweb tiers were more likely to be proficient on the grade 3 
Pennsylvania System of School Assessment 
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AlMSweb assessment grade 


Percentage proficient on grade 3 PSSA in 
ELA 


Mlier 1:Attarget OTier 2: Strategic intervention Tier 3: Intensive intervention 


ELA is English language arts. PSSA is Pennsylvania System of School Assessment. 

Note: Analysis was based on random sample of students used to set the threshold and was restricted to students who completed the PSSA and AlMSweb: 
4,174 to 5,266 students had information on the grade 3 PSSA and AlMSweb in grades K to 3. Analysis was not weighted. 

Source: Authors’ analysis based on SDP 2014—2018 data described in appendix A. 
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On the spring AlMSweb in grades K through 3, the cutoff for the Tier 1 grade-level target correctly predicted 52 to 
78 percent of students who were not proficient in ELA on the grade 3 PSSA and 65 to 83 percent of students who 
were proficient. The Tier 1 cutoff on the AlMSweb in the spring of kindergarten correctly predicted fewer students 
who were not proficient on the grade 3 PSSA than the study-selected KEI threshold of 6 (52 versus 73 percent) but 
predicted more of the students who were proficient (83 versus 53 percent) (table 7). 


In contrast, by spring of grade 2, the Tier 1 cutoff correctly predicted 76 percent of students who were not 
proficient on the grade 3 PSSA and 81 percent of students who were proficient (table 7). This further demonstrates 
the increased accuracy of the AlMSweb for predicting reading proficiency assessed at closer points in time. 


Table 7. The AlIMSweb Tier 1 cutoff in later grades correctly predicted the proficiency status of more students 
than the AlMSweb Tier 1 cutoff in kindergarten 
Correct predictions for: 
PNosxo) [hn 
difference 
between actual 


and predicted Noy 
Actual acre llancte| proficiency Proficient proficient Number 
Assessment proficiency proficiency (percentage students students All students of 
andthreshold (percentage) (percentage) rexey[alky) (percentage) (percentage) (percentage) students 


KEI index threshold (fall of kindergarten) 


6.00 37 37 1 53 73 66 3,521 
7.00 37 17 21 29 91 68 3,521 


AlMSweb tier 1 cutoff (spring of each grade) 


Kindergarten 37 61 23 83 52 64 4,696 
Grade 1 36 42 6 65 71 69 4,174 
Grade 2 37 45 8 81 76 77 5,027 
Grade 3 37 42 5 76 78 77 5,266 


Note: The table is based on the sample used to select the threshold and the AlMSweb data was not weighted. The KEI data was weighted to represent all 
students with KEI scores. The difference between actual and predicted proficiency may not equal the difference between the actual and predicted proficiency 
columns in the table due to rounding. 

Source: Authors’ analysis based on School District of Philadelphia data 2014—2018 data described in appendix A. 


The AlMSweb tier cutoffs in the spring of kindergarten through grade 3 were not designed as an indicator of the 
overall percentage of students who are on-track for reading proficiency and were less accurate than the study- 


selected KEI threshold of six from the fall of kindergarten for predicting the overall percentage of children 
proficient in reading by grade 3 (table 7). 


Limitations 


Three limitations should be kept in mind. First, the study focuses ona single cohort of students in SDP—those who 
entered kindergarten in the 2014/15 school year—two-thirds of whom had KEI scores. The threshold might or 
might not yield the same accuracy when predicting grade 3 reading proficiency of future cohorts of students in 
SDP, who may have more complete data on KEI scores and will take the PSSA in a different school year. The study 
also does not include students in Philadelphia charter and private schools, or students from other districts in 
Pennsylvania, so the accuracy of the threshold might also differ for these groups of students. Also, changes in KEI 
administration or instruction and supports for students between kindergarten and grade 3 could generate 
different relationships than those described. SDP has improved teacher training on the KEI administration and has 
created initiatives intended to increase reading supports to struggling students between kindergarten and grade 
3, such as providing instructional coaches focused on early literacy and strengthening classroom literacy 
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environments. These additional supports could alter the relationship between KEI and PSSA scores in ELA. 
Likewise, changes in the AlMSweb assessment (such as the change to AlIMSwebPLUS in the 2018/19 school year) 
could alter the relationships between the KEI and PSSA and those scores. 


Second, to account for students who were missing scores on the PSSA in ELA when calculating whether 
kindergarteners are on-track, the study reweighted the analyses based on students’ KEI scores and demographic 
characteristics. This assumes that students who were missing PSSA scores in grade 3 would have received the 
same scores as those with similar KEI scores and background characteristics in kindergarten. However, students 
who were missing PSSA scores in grade 3 might have had different scores than assumed. For example, students 
who moved out of SDP might have characteristics that were not measured in the data (such as family instability), 
which could reduce their achievement by grade 3.2 Students who were missing KEI scores were less likely to be 
White than all kindergarteners. For students who were missing KEI scores, limited data (only background 
characteristics) were available for predicting their likely proficiency in reading in grade 3. Thus, the study’s main 
analyses did not make predictions for students who are missing KEI scores. 


Third, this study only analyzed the two dimensions of the Pennsylvania KEI—Emerging Academic Competencies 
and Learning Engagement Competencies—for which the study team received data. These dimensions were 
validated in a study of the KEI (Howard et al., 2017). However, it is possible that particular KEI items within these 
dimensions (for example, early reading skills in the Emerging Academic Competencies dimension) or items outside 
of these dimensions (for example, language comprehension skills) might have even stronger correlations with 
grade 3 PSSA scores in ELA. Alternative assessments, such as the fall kindergarten AlMSweb used by SDP or 
kindergarten entry assessments used by other states, might have different relationships with reading proficiency 
in grade 3. For example, other states’ kindergarten entry assessments could have stronger relationships with later 
reading proficiency if they more accurately measure skills that are more predictive of reading proficiently or if 
states train teachers to rate students’ skills more accurately (Jenkins, Hudson, & Johnson, 2007). 


Implications 

SDP can apply the study-established threshold of six on a 1-10 index based on two KEI dimensions (Emerging 
Academic Competencies and Learning Engagement Competencies) to the KEI scores of future kindergarten 
cohorts, allowing SDP to examine changes over time in the percentages of students who are on-track for reading 
proficiently in grade 3. Differences in this indicator for future cohorts might reflect the efforts of Philadelphia’s 
Read by 4th partnership’s citywide investments in early childhood education from birth to age 5, such as parent 
education about reading to children and the expansion of prekindergarten in the city. Differences in this indicator 
could also reflect changing composition of the student population, such as changing demographics. 


SDP could also use data on more recent cohorts of kindergarteners to reassess the accuracy of and potentially 
update the threshold for the indicator of whether entering kindergarteners are on-track for reading proficiently. 
The study’s analyses were based on approximately two-thirds of kindergarteners in SDP who had KEI scores in 
2014, the first year the district implemented a new version of KEI. SDP staff indicated that the KEI was more 
consistently administered in recent years. Moreover, should KEI data become available for children in charter and 
private schools, the analyses could be extended to those groups. 


SDP may also wish to use a different threshold on the KEI, AlIMSweb scores, other measures, or some combination 
to predict which individual students are at risk of not being proficient in reading, as opposed to predicting cohort- 
level proficiency rates. Previous research has suggested selecting thresholds that correctly identify 90 percent of 
students who are at risk of not being proficient (Compton et al. 2010, Jenkins et al. 2007). Yet, in early grades, 


* Students might also be missing PSSA scores because they were absent on the assessment date or were not promoted to 
grade 3. However, SDP staff indicated that few students are retained in earlier grades. 
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such thresholds often have low rates of correctly identifying students who will be proficient, requiring decisions 
about the appropriate tradeoff between correctly identifying students who will and will not be proficient (Hintze, 
Ryan, & Stoner, 2003; O’Connor & Jenkins, 1999). Consistent with findings from previous research, the threshold 
of 7 on the 1-10 KEI index correctly predicted over 90 percent of students who will not be proficient on the grade 
3 PSSA in ELA while correctly predicting 29 percent of students who will be proficient. The Tier 1 grade-level targets 
on the spring AlMSweb in kindergarten correctly predicted 52 percent of students who will not be proficient, but 
83 percent of students who will be proficient. In later grades, the Tier 1 grade-level targets correctly predicted 
between 71 and 78 percent of not proficient students, but had overall correct prediction rates above both KEI 
thresholds. SDP’s decisions about which threshold or measures to use will depend on how the district balances 
the costs of identifying students who truly are not at risk versus the costs of not identifying students who truly are 
at risk (Elliot, Haui, & Roach, 2007). 


The study also highlights the importance of SDP frequently reassessing students’ reading skills at early ages to 
determine which students need additional reading supports. Students’ reading skills change markedly between 
entering kindergarten and grade 3, as students acclimate to school and learn from their teachers, parents, and 
peers over multiple school years (O’Connor & Jenkins, 1999). In fact, KEI scores from the beginning of kindergarten 
had weaker correlations with grade 3 PSSA scores than AlMSweb scores at the end of kindergarten, and the 
strength of correlations between AlMSweb scores and grade 3 PSSA scores generally increased from kindergarten 
to grade 3. 


SDP could consider developing other indicators to identify whether individual students need support in reading 
or to measure the percentages of students who are on-track for reading proficiently at the end of kindergarten 
through grade 3. Including multiple measures from the AlMSweb or KEI could reduce errors in the predictions for 
identifying individual students who need supports. 


Future research could also explore creating indicators of the percentages of students who are on-track for reading 
proficiently based on the AlMSweb. For example, SDP could use successive spring AlMSweb scores to predict the 
percentages of students at the end of kindergarten through grade 2 who are on-track for grade 3 PSSA proficiency. 
Or, SDP could use AlMSweb scores to create indicators that measure cohorts’ changes in reading skills from year 
to year. For example, they could measure the percentage of students in grade 1 who are on-track to be at the 
AlMSweb grade-level target by the end of grade 2. That research would further inform ongoing district efforts to 
support children’s reading achievement growth from kindergarten through grade 3 by identifying specific times 
in students’ academic careers that may disproportionately predict their likelihood of reading proficiency. 


This study provides a basis for the use of kindergarten entry assessments that other states could consider. Many 
other states are in the process of understanding how to use kindergarten assessments to inform instruction, 
identify students in need of additional support, and measure children’s school readiness and progress in early 
grades (Shields, Cook, & Greller, 2016). The moderate correlations between KEI Emerging Academic Competencies 
and ELA scores on the grade 3 PSSA (0.43) are in line with past literature on the relationship between children’s 
early skills and later academic achievement (Duncan et al., 2007; LaParo & Pianta, 2000). These links between 
teacher-reported early learning skills and children’s reading proficiency in later grades support the use of 
kindergarten entry assessments as an aggregate indicator of children’s later skills. They also potentially support 
the use of these assessments as an early indicator for identifying individual students who need additional support, 
though this may be more error-prone than an indicator of the overall proportion of students expected to be 
proficient. In combination with additional assessments, an early indicator could bring earlier reading intervention 
services to improve students’ chances of becoming proficient readers. 
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Appendix A. Methods 


This appendix describes the data, samples, weights, and analysis methods for the Regional Educational Laboratory 
(REL) Mid-Atlantic study to identify a school readiness indicator for the School District of Philadelphia (SDP). 


Data 

The data used for this study came from SDP’s administrative database and included (1) students’ school 
enrollment records and demographic characteristics from the 2014/15 school year, (2) Kindergarten Entry 
Inventory (KEI) scores for students who entered kindergarten in 2014, (3) Pennsylvania System of School 
Assessment (PSSA) in English language arts (ELA) scores for students who took the grade 3 PSSA assessment in 
2018, and (4) AlMSweb Reading scores for each spring semester from 2015 (kindergarten) to 2018 (grade 3). 


School enrollment records and demographic characteristics 

This study used enrollment records for students enrolled in SDP in kindergarten at any point between September 
and December 2014, the period when KEI assessments were conducted for the 2014/15 school year. The 
characteristics used to describe the study samples and construct weights for the analysis are listed in table A1. 


Table A1. Student characteristics used in the study 


Characteristic Description 


Female Whether student was female 


Race/ethnicity Whether student was 


e —- Black/African American 
e ~—_ Hispanic/Latino 
e White 
e = Multiracial/other 
Economically disadvantaged Whether student participates in Supplemental Nutrition Assistance Program, Temporary 
Assistance for Needy Families, Medicaid, or other social service programs and is therefore 
eligible for free lunch without additional verification 


English learner Whether student was an English learner or had limited English proficiency 
Special education Whether student received special education services 


Source: Authors’ analysis based on School District of Philadelphia 2014/15 data. 


Kindergarten Entry Inventory 

The KEI is a teacher-administered assessment created by Pennsylvania’s Office of Child Development and Early 
Learning to understand and track students’ skills at kindergarten entry (Pennsylvania Office of Child Development 
and Early Learning, 2017). The full KEI includes 30 indicators across five areas: social and emotional development; 
English language arts; mathematics; approaches to learning; and health, wellness, and physical development. On 
each of these indicators, teachers give students a rating of 1 (Not yet evident), 2 (Emerging), 3 (Evident), or 4 
(Exceeds) based on their observations of students in the classroom. 


The study analyzed two KEI dimensions identified as having evidence of validity by a previous study (Howard et 
al., 2017). For each dimension, the score is the average rating (on a 1- to 4-point scale) across the indicators: 


e Emerging Academic Competencies score is the average rating across 12 indicators of early literacy and early 
mathematics skills. 


e Learning Engagement Competencies score is the average rating across nine indicators of behavioral and motor 
skills. 


The KEI indicators included in each dimension and those excluded from the study because they were not part of 
a validated dimension are listed in table A2. 
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Table A2. Kindergarten Entry Inventory indicators, by dimension 
Included in 


Dimension this study Tae | (ek=} Coy 3 


Emerging Academic Competencies Yes Counting 

Identifying shapes 

Naming numbers 

Operations and algebraic thinking 
Phonics 

Phonological awareness 

Print concepts/letters 

Print concepts/words 

Stages of writing 

Text analysis 

Text structure 

Writing process 

Behavior regulation 

Conflict resolution 

Control and coordination: fine motor 
Control and coordination: gross motor 
Emotional regulation 

Engagement, attention, and persistence 
Reasoning and problem solving 
Self-awareness 

Task analysis 

Collaborative communication 
Conventions of English language 
Expressive language 

Receptive language 

Positional words 

Curiosity and initiative 

Data 

Measurement 

Stages of play 


Learning Engagement Competencies Yes 


Not part of a validated dimension No 


000000@0UdCM0OUOCNUODUCOOUCOUUCOUCUCOUCOCONWC COC ONOLUCOCUCOCOCOCO CO COCO COCO CO 


Source: Adapted from Howard et al., 2017. 


Pennsylvania System of School Assessment 
The PSSA is Pennsylvania’s annual statewide standardized assessment, which covers both math and ELA in grades 
3 through 8. This study focused on the grade 3 PSSA in ELA. PSSA scale scores ranged from 600 to 1551, and the 
performance levels can be classified as proficient (proficient or advanced) and not proficient (basic or below basic) 
(Pennsylvania Department of Education, 2018). 


AlMSweb 

AlMSweb is a widely used progress-monitoring system (Pearson, 2019). The system allows districts to select from 
a range of established assessments (or develop their own assessments) and track student performance against 
benchmarks. SDP used the grade-appropriate literacy-related AlMSweb assessments for each semester from 
spring 2015 through spring 2018 (table A3). 
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Table A3. School District of Philadelphia’s use of AlMSweb spring reading assessments, by grade 


Kindergarten Grade 1 Grade 2 CC] ¢-10(=5e} 
(2014-15) (2015-2016) (2016-2017) (2017-2018) 
Letter naming fluency X 
Nonsense word fluency X X 
Reading curriculum-based x 
measurement of oral reading 
fluency 


Source: School District of Philadelphia 2014—2018 data. 
Table reads: School District of Philadelphia used the AlMSweb letter naming fluency assessment to assess students in kindergarten. 


The AlMSweb data included two measures of student performance. Raw scores are the number of correct 
answers. Performance tiers describe students’ performance relative to the assessments’ benchmarks: at target 
(Tier 1), requiring strategic intervention (Tier 2), or requiring intensive intervention (Tier 3). 


Samples 


The study sample was the cohort of kindergarteners who entered SDP by December of the 2014/15 school year 
(13,335 students). The samples used to answer the research questions are smaller because students needed to 
have data on the relevant assessments. As discussed in the report, 2014 was the first year that the entire district 
was required to use the KEI, and 58 to 68 percent of kindergarteners had KEI scores for one or both dimensions 
(table A4). District staff provided several explanations for missing KEI scores.? First, the 2014/15 school year was 
the first year that the entire district was required to use the KEI, so some teachers may not have complied with 
the requirement. Second, students must be rated on all individual indicators included in each KEI dimension to 
receive a score, and some students were missing data on particular indicators. Third, SDP experienced some issues 
with matching students based on the identifiers included in the data collected by the state (KEI and PSSA) and SDP 
(enrollment records and AlMSweb). By grade 3, about 40 percent of the full sample of kindergarteners and 65 
percent of those with KEI scores had ELA scores on the grade 3 PSSA. Missing PSSA scores could be due to students 
moving out of the district, transferring to non-district schools (Philadelphia has a large charter sector), being 
absent on the assessment date, or repeating a grade. The sample of students with KEI and PSSA scores includes 
43 percent of all students enrolled in an SDP school for grade 3 during the 2017/18 school year. 


Table A4. Two-thirds of kindergarteners had Kindergarten Entry Inventory scores and about 40 percent had 
both Kindergarten Entry Inventory and Pennsylvania System of School Assessment scores 


With scores on the KEI dimension With scores on the KEI dimension and PSSA scores 


As percentage of 
students with 


Number of Percentage of Number of Percentage of scores on the KEI 
KEI dimension students kindergarteners students kindergarteners dimension 
Emerging Academic 8,089 61 5,267 39 65 
Competencies 
Learning Engagement 9,089 68 5,944 45 65 
Competencies 
Both KEI dimensions 7,707 58 5,029 38 65 


KEI is Kindergarten Entry Inventory. PSSA is Pennsylvania System of School Assessment. 

Note: 13,335 kindergarteners entered School District of Philadelphia by December of the 2014/15 school year. 

Source: Authors’ analysis based on School District of Philadelphia 2014/15 and 2017/18 data. 

Table reads: 8,089 kindergarteners (61 percent of all kindergarteners) had scores on Emerging Academic Competencies dimension. 5,267 kindergarteners 
(39 percent of all kindergarteners and 65 percent of those with scores on the Emerging Academic Competencies dimension) had scores on both the Emerging 
Academic Competencies dimension and grade 3 PSSA. 


3 Personal communication on April 22, 2019. 
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The sample sizes for analyses used to answer each research question that are presented in the main text (table 
AS) differ for two key reasons: 


e Students missing data on one or more assessments. Some students were missing data on one or both of the 
KEI dimensions, grade 3 PSSA in ELA, or spring AlIMSweb assessments in grades K through 3. 


e Analyses that focus on a random 70 or 30 percent of the sample. Following the typical approach for predictive 
analyses, the study team split the sample randomly into two parts, conducting analyses on the first part to 
select the threshold (70 percent of the full sample) and then checking the accuracy of the predictions using the 
second part (30 percent of the full sample). To ensure that the two parts would be similar in terms of students’ 
PSSA proficiency, the data were stratified into three groups based on student’s PSSA proficiency level 
(proficient, not proficient, and missing PSSA data) before students were randomly split into the two parts. 
Consistent with random sampling, students in the two parts did not differ on any characteristics by more than 
5 percent of a standard deviation for all students (table A6). For comparison with findings from research 
question 1 (which establish the feasibility of setting a threshold in a subsample of data), research question 3 
also used a 70 percent sample (so its findings can be used to inform the feasibility of setting a threshold based 
on different assessments in future work). 


Table AS. Sample sizes by research question and analysis sample for the analyses presented in the main text 


Research question (RQ) Analysis sample Number of students 
RQ1. Relationship between KEI scores and grade 3 Random 70 percent sample of students 3,521 
PSSA in ELA scores (in sample used to select who took the KEI in 2014/15 and the grade 
threshold) 3 PSSA in ELA in 2017/18 
RQ2. Accuracy of threshold (in sample used to Random 70 percent sample of students 3,521 
select it) who took the KEI in 2014/15 and the grade 

3 PSSA in ELA in 2017/18 
RQ2. Accuracy of threshold (in sample not used to Random 30 percent sample of students 1,508 
select it) who took the KEI in 2014/15 and the grade 

3 PSSA in ELA in 2017/18 
RQ3. Relationship between KEI scores, spring Random 70 percent sample of students 3,689-5,446 
AlMSweb Reading scores in grades K-3, and grade 3. __ who took both of the assessments being 
PSSA in ELA analyzed in each year (that is, AIMSweb and 


KEI or AlMSweb and PSSA) 


ELA is English language arts. KEI is Kindergarten Entry Inventory. PSSA is Pennsylvania System of School Assessment. 

Note: The number of students varied based on the KEI dimension (Emerging Academic Competencies, Learning Engagement Competencies, and both 
dimensions) and other assessments (AlMSweb Reading or grade 3 PSSA in ELA) included in the analyses. 

Source: Authors’ analysis based on School District of Philadelphia 2014—2018 data. 
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Table A6. Students whose data were used to set the threshold and those whose data were used to assess the 
threshold’s accuracy had similar characteristics 


Students whose 
data were used to 


Students whose data assess the Difference 

were used to set the threshold’s accuracy Standard (in standard 

threshold (mean? or (mean? or deviation for deviation 
Characteristic percentage’) percentage®) all students units) 
Female 49 48 50 0.020 
Black/African American 49 51 50 0.040 
Hispanic/Latino 21 20 40 0.025 
White 14 14 35 0.000 
Multiracial or other race/ethnicity 16 15 36 0.028 
Economically disadvantaged program 
participant 67 66 47 0.021 
English learner student 10 10 30 0.000 
Special education student 8 8 37 0.000 
Emerging Academic Competencies score 2.41 2.39 0.66 0.030 
Learning Engagement Competencies score 2.50 2.51 0.54 -0.019 
Kindergarten spring AlMSweb score 47.8 47.5 20.7 0.014 
Grade 1 spring AlIMSweb score 57.1 56.4 34.3 0.020 
Grade 2 spring AlIMSweb score 84.4 84.8 43.7 -0.009 
Grade 3 spring AlIMSweb score 104.6 103.9 49 0.014 
Grade 3 PSSA score 973 974 99 -0.010 
Grade 3 PSSA proficiency 34 34 48 0.000 
Number of students 5,435-9,317 2,322-4,018 


aThe values for Emerging Academic Competencies scores, Learning Engagement Competencies scores, AlMSweb scores, PSSA scores are mean values. 
bThe values for female, Black/African American, Hispanic/Latino, White, Multiracial or other race/ethnicity, economically disadvantaged program 
participant, English language student, special education student, grade 3 PSSA proficiency are percentages. 

PSSA is Pennsylvania System of School Assessment. 

Note: Descriptive statistics were not weighted. 

Source: Authors’ analysis based on School District of Philadelphia 2014—2018 data described in appendix A. 


Weights 

This section describes the weights used in the analyses to account for missing data and the differences between 
the characteristics of students included in the analysis and those in the population of interest. It also describes 
how the study assessed differences between the analytic sample and the population of interest. 


Primary weights 


SDP intends to apply the threshold to all entering kindergarteners with KEI scores to estimate the proportion of 
students on-track to achieve reading proficiency in grade 3. However, not all of these students will take the grade 
3 PSSA in ELA, even though the analyses that set the threshold are based on students with both KEI and grade 3 
PSSA scores. Thus, the analyses for research questions 1 and 2 used weights to make the findings based on 
students who have KEI and grade 3 PSSA scores representative for all kindergarteners with KEI scores. 
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The analyses for research questions 1 and 2 used trimmed inverse probability weights, which were constructed 
following the National Assessment of Education Progress (2017) procedure. First, a logistic regression model 
estimated each student’s probability of taking the grade 3 PSSA in ELA. This model included students’ KEI scores 
on the two domains; students’ characteristics (gender, race/ethnicity, economic disadvantage, English learner, 
and special education); and indicators for missing KEI scores or student characteristics. It also included interactions 
between race/ethnicity indicators and KEI scores (and indicators for missing KEI scores) because multiple 
unweighted analytic samples had differences on these characteristics. 


Second, the initial weights were set equal to the inverse of students’ predicted probability of taking the PSSA from 
the logistic regression. Therefore, students who represent the groups that were least likely to have PSSA data 
received greater weight in the analysis. 


Finally, the initial weights were “trimmed” to ensure that no single student would have too much influence over 
final estimates. The study team used a maximum weight of 3.5 times the median weight, following the National 
Assessment of Education Progress (2017) guidelines. Students with calculated weights from the logistic model 
that were larger than 3.5 were given a weight of 3.5. 


For research question 3, the analysis was exploratory and not intended to set an indicator that could be applied 
to a representative population of students (such as all students with AlMSweb scores in a particular grade). Thus, 
to facilitate comparisons across the analyses in research question 3, these analyses did not use weights. 


Alternative weights 


To test the sensitivity of the findings to the weighting strategy, the analyses to address research question 2 were 
repeated using two types of alternative weighting approaches: one that weighted the sample to represent all 
entering kindergarteners in SDP and one that was not weighted. 


SDP would like to understand the proportion of all entering kindergarteners who are on-track to achieve reading 
proficiency in grade 3, but 30 to 40 percent of kindergarteners were missing KEI scores, a key predictor of grade 3 
PSSA scores in ELA. Therefore, as a sensitivity test, another weight was constructed to calculate a predicted 
probability of proficiency that was reflective of all kindergarteners. This weight used a two step-approach, which 
incorporated the available KEI data. First, a logistic regression model was used to estimate each student’s 
probability of having a KEI score in fall 2014. This model included students’ demographic characteristics and 
indicators for missing demographic characteristics. Second, the predicted probability of having a KEl score was 
multiplied by the predicted probability of having a grade 3 PSSA in ELA score, conditional on KEI scores and other 
characteristics (as calculated in the logistic regression model described above). Next, as above, the initial weight 
was set to the inverse of the product of the two predicted probabilities and trimmed to a maximum of 3.5 times 
the median weight. 


Assessing differences between samples 

This section presents analyses that assess the extent to which weighted and unweighted analytic samples were 
similar to other samples and populations of interest, based on the characteristics available in the SDP data. It also 
provides analyses that examine how differences in characteristics relate to the primary outcome of interest— 
students’ grade 3 PSSA in ELA scores. These analyses provide context about the extent to which the findings for a 
given analytic sample might generalize to other groups of students. Analytic samples were defined in terms of 
whether students (1) were enrolled in SDP in kindergarten in the fall of 2014, (2) had KEI scores, and (3) had scores 
on the grade 3 PSSA in ELA. Samples were considered similar on a given characteristic if, on average, differences 
between groups did not exceed 5 percent of a standard deviation for all students. However, differences above 5 
percent of a standard deviation could still represent small percentage point differences. 
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Differences between unweighted samples of all kindergarteners and students with KEI scores. For the indicator to 
accurately measure the percentage of a cohort of entering kindergarteners who are on track for reading 
proficiently, ideally, students with scores on the KEI must not be substantially different from those without scores. 
Without weighting, the students enrolled in kindergarten by December 2014 and the subsamples of students with 
KEI and KEI and PSSA in ELA data were similar on most characteristics (table A7). However, there were a few 
differences that exceeded 5 percent of a standard deviation. For example, students with both KEI dimension 
scores were more likely to be White than all kindergarteners. In addition, students with both KEI and PSSA in ELA 
scores were less likely to be Black or receiving special education services than all kindergarteners. The analyses 
only examined the characteristics available in the data for all kindergarteners, so they do not provide information 
about differences in students’ achievement. 


Table A7. Among students whose data was used to set the threshold, those with Kindergarten Entry Inventory 
data and those with Kindergarten Entry Inventory and Pennsylvania System of School Assessment data 
differed from all kindergarteners on some demographic characteristics 


Subsample with both KEI Subsample with PSSA and both 
All kindergarteners dimension scores KEI dimension scores 


Difference from Difference from 
S\r-Taler-lae| all all 
deviation kindergarteners kindergarteners 
Characteristic Percentage ‘(S32)) Percentage (in SD units) Percentage (in SD units) 
Female 48 50 48 0.011 49 0.031 
Black/African American 49 50 48 -0.035 46 -0.072T 


Hispanic/Latino 21 41 22 0.036 
White 14 35 16 0.053t 15 0.026 
Multiracial or other race/ 16 36 17 0.023 17 0.035 
ethnicity 

Economically 66 47 68 0.044 68 0.039 
disadvantaged program 


participant 
English language learner 10 30 10 0.013 11 0.045 


Special education student 8 27 8 -0.015 5 -0.105t 


Number of students 9,317 


+ Larger than 5 percent of a standard deviation. 

KEI is Kindergarten Entry Inventory. PSSA is Pennsylvania System of School Assessment. 

Note: Descriptive statistics were not weighted. 

Source: Authors’ analysis based on School District of Philadelphia 2014/15 and 2017/18 data described in appendix A. 


Differences between the characteristics of unweighted samples of students with KEI scores and students with both 
KEI and grade 3 PSSA ELA scores. The analytic sample of students with KEI data had several characteristics that 
differed by more than 5 percent of a standard deviation from those of students with both KEI and grade 3 PSSA in 
ELA scores (table A8). For example, compared with students with KEI scores, students with both KEI and PSSA 
scores had higher KEI scores on both dimensions, were more likely to be Hispanic/Latino, and were less likely to 
receive special education services. 
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Table A8. Among students whose data were used to set the threshold, Kindergarten Entry Inventory scores 
and other student characteristics differed for students with Kindergarten Entry Inventory and grade 3 
Pennsylvania System of School Assessment in English Language Arts scores and students with just 
Kindergarten Entry Inventory scores 


Kindergarteners with both KEI Subsample with PSSA and both KEI dimension 
dimension scores scores 


DYhaiclaclalecminelan 


Mean? or S\rlaler-lae| Mean? or kindergarteners with KEI 
Characteristic percentage? deviation (SD) percentage> scores (in SD units) 
Female 48 50 49 0.020 
Black/African American 48 50 46 -0.038 
Hispanic/Latino 20 40 22 0.060t 
White 16 37 15 -0.026 
Multi-racial or other race/ethnicity 17 37 17 0.012 
Economically disadvantaged program 
participant 68 47 68 -0.006 
English learner student 10 30 14 0.031 
Special education student 8 26 5 -0.093t 
Emerging Academic Competencies 
score 2.395 66 2.440 0.068t 
Learning Engagement Competencies 
score 2.515 54 2.552 0.069t 
Number of students 5,398 3,521 


The values for Emerging Academic Competencies scores and Learning Engagement Competencies scores are mean values. 

’The values for female, Black/African American, Hispanic/Latino, White, Multiracial or other race/ethnicity, economically disadvantaged program participant, 
English learner student and special education student are percentages. 

+ Larger than 5 percent of a standard deviation. 

KEI is Kindergarten Entry Inventory. PSSA is Pennsylvania System of School Assessment. 

Note: Descriptive statistics were not weighted. 

Source: Authors’ analysis based on School District of Philadelphia 2014/15 and 2017/18 data described in appendix A. 


Relationships between student characteristics and grade 3 PSSA in ELA scores. The threshold was set using data 
on students with both KEI and grade 3 PSSA in ELA scores, but it will be ultimately applied to students with KEI 
scores (some of whom will not have PSSA scores in grade 3). Findings for students with both KEI and grade 3 PSSA 
scores might not generalize to students with KEI scores if there are substantial differences between the two groups 
of students on characteristics that are correlated with grade 3 PSSA scores that are not adjusted using weights. 


KEI scores had weak to moderate correlations with grade 3 PSSA scores (table A9). Other student characteristics 
(such as economic disadvantage) had weak but statistically significant correlations with grade 3 PSSA scores. 
Therefore, the analyses used weights to make the sample of students with both KEI and grade 3 PSSA scores 
representative of students with KEI scores. 
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Table A9. Among students whose data were used to set the threshold, Kindergarten Entry Inventory scores 
had weak to moderate correlations with grade 3 Pennsylvania System of School Assessment scores, while 
other student characteristics had weak correlations with grade 3 Pennsylvania System of School Assessment 
scores 


Characteristic Correlation with grade 3 PSSA scores 


Female 0.11*** 
Black/African American -0.21*** 
Hispanic/Latino -0.11*** 
White 0.24*** 
Multiracial or other race/ethnicity 0.20*** 
Economically disadvantaged program participant -0.20*** 
English learner student -0.04** 

Special education student -0.12*** 
Emerging Academic Competencies score 0.42*** 
Learning Engagement Competencies score 0.33*** 
Number of students 3,729-5,311 


* Significant at p = .05; ** significant at p = .01; *** significant at p = .001. 

PSSA is Pennsylvania System of School Assessment. 

Note: Correlations were not weighted. 

Source: Authors’ analysis based on School District of Philadelphia 2014/15 and 2017/18 data described in appendix A. 


Differences between the characteristics of the weighted samples of students with KEI scores and students with 
both KEI and grade 3 PSSA in ELA scores. Weighting reduced the differences between students with both KEI and 
grade 3 PSSA in ELA scores and students with KEI scores (table A10). Between the weighted analytic samples, no 
characteristics had a difference greater than 5 percent of a standard deviation. 
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Table A10. Among students whose data were used to set the threshold, the weights used in the main analyses 
make the students with Kindergarten Entry Inventory and Pennsylvania System of School Assessment scores 
similar to students with just Kindergarten Entry Inventory scores 


Kindergarteners with both KEI Weighted subsample with PSSA and both 
dimension scores KEI dimension scores 


Difference from students 


Mean? or Standard deviation Mean? or with both KEI scores (in 
Characteristic percentage> ‘S32)) percentage? SD units) 
Female 48 0.500 48 0.001 
Black/African American 48 0.499 48 0.002 
Hispanic/Latino 20 0.399 20 0.001 
White 16 0.367 16 0.000 
Multiracial or other race/ ethnicity 17 0.371 16 -0.003 
Economically disadvantaged program 68 0.467 68 -0.001 
participant 
English learner student 10 0.304 10 0.001 
Special education student 8 0.264 8 -0.002 
Emerging Academics Competencies score 2.395 0.659 2.398 0.005 
Learning Engagement Competencies score 2.515 0.536 2.519 0.008 
Number of students 5,398 3,521 


*The values for Emerging Academic Competencies scores and Learning Engagement Competencies scores are mean values. 

>The values for female, Black/African American, Hispanic/Latino, White, Multiracial or other race/ethnicity, economically disadvantaged program participant, 
English learner student and special education student are percentages. 

KEI is Kindergarten Entry Inventory. PSSA is Pennsylvania System of School Assessment. 

Note: None of the differences were larger than 5 percent of a standard deviation. 

Source: Authors’ analysis based on School District of Philadelphia 2014/15 and 2017/18 data described in appendix A. 


Methods 
This section describes the methods used to answer each of the study’s research questions. 


Research question 1: What is the relationship between KEI scores and scores on the grade 3 PSSA in ELAP 


Descriptive statistics and pairwise correlations were used to assess the relationship between each KEI dimension 
score and students’ PSSA scaled score among students whose data were used to set the threshold. Positive 
correlations were classified as follows: weak (0.1-0.39), moderate (0.4—0.69), strong (0.7—99), and perfect (1; 
Dancey & Reidy, 2007). Statistical significance was evaluated at and below the 0.05 level. 


Logistic regressions were used to examine the relationship between the KEI dimensions and grade 3 PSSA in ELA 
proficiency (equation A1). These regressions analyzed KEI dimensions together (as in equation A1) or separately 
(by excluding the other dimension). Logistic regressions are designed to analyze binary (yes/no) outcomes like 
proficiency, and their results can be translated into readily interpretable findings (odds ratios or average marginal 
effects). For example, for Emerging Academic Competencies, the coefficient (B,,.) in the logistic equation is the 
log odds, and the odds ratio is ePac. The coefficients and odds ratios represent the average relationship between 
ELA proficiency and KEI scores. Average marginal effects, which can be interpreted as the average increase in the 
probability of proficiency for every 1-point increase, were calculated for the report. McFadden’s Pseudo R Squared 
is a measure of how well the model fits the data compared to a model with no predictors. SDP wanted to set a 
single threshold for all groups of students so the regressions did not include student characteristics as covariates. 


REL Mid-Atlantic | Using kindergarten entry assessments 27 


Equation A1. Logistic regression model for relationship between Kindergarten Entry Inventory dimensions and 
grade 3 Pennsylvania System of School Assessment proficiency 


e” + BracEAC;+ BpgcLEC, 


Pr(Y, =1) 


~ 1+ ef ot BracEAC;+ Brac LEC, 


bf = Student is proficient on the grade 3 PSSA in ELA 
EAC, = Student’s Emerging Academic Competencies score 


LEC, = Student’s Learning Engagement Competencies score 


Research question 2: What threshold scores on the KEI most accurately predict reading proficiency on the grade 3 
PSSA in ELA for the cohort? What threshold score on the KEI has a correct prediction rate of at least 90 percent 
for the individual students who were not proficient in ELA on the grade 3 PSSA? 


Answering research question 2 involved creating an index based on the two KEI dimensions, setting the threshold 
based on a subset of data, and assessing the accuracy of the threshold in data that were not used to set the 
threshold. 


Creating an index based on the two KEI dimensions. The index of the two KEI dimensions (equation A2) was a 
weighted average based on the coefficients from the logistic regression shown in equation A1. This index is scaled 
to range from 1 to 10, such that if students performed perfectly (were rated as “Exceeds” on all indicators on both 
dimensions) they would receive a 10. This scaling was used because SDP wanted to prevent the index from being 
confused with a raw KEI score (range: 1 to 4) or a percentile rank (range: 1 to 100). 


Equation A2. Formula for calculating a Kindergarten Entry Inventory index score 


. gh Bec EAG + BygcLEC, Min) _ 1, D(LAATEAC, + 0.108LEC, ~1.255) _ 
Max — Min 5.020—1.255 


1+9(1.147EAC, + 108LEC, a) 
3.765 


A = Student’s Kindergarten Entry Inventory index score 
EAC, = Student’s Emerging Academic Competencies score 


LEC, = Student’s Learning Engagement Competencies score 

Pits = Estimated coefficient on Emerging Academic Competencies score from equation Al = 1.147 

Pr = Estimated coefficient on Learning Engagement Competencies scores from equation A1 = 0.108 

Min = Minimum estimated value of Bee AG, + Brae LEG, , Which corresponds to EAC, =1 and LEC, =1. 
Min =1.147+0.108 =1.255 

Max = Maximum estimated value of Bap PAG, + Bic LEC,, which corresponds to EAC, =4 and LEC, =4. 
Pe oad 

Max = (1.147*4+0.108* 4) = 5.020 


Setting the threshold based on a subset of data. Several statistics were calculated to assess the accuracy of 
potential KEI thresholds and select a threshold (Box A1). In the sample used to select the threshold, the percentage 
of students correctly predicted to be proficient and the percentage of students correctly predicted to be not 
proficient were calculated for all possible thresholds on (1) Emerging Academic Competencies, (2) Learning 
Engagement Competencies, and (3) the index of Emerging Academic Competencies and Learning Engagement 
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Competencies. Then, the percentage of correct predictions for students who were and were not proficient based 
on these thresholds across the different potential indicators were compared to decide which dimensions to use 
for setting the threshold (shown in figure 2 in the report). 


Box A1. Statistics used to assess the accuracy of potential Kindergarten Entry Inventory thresholds 


Absolute difference between the actual proficiency rate and predicted proficiency rate 
based on the threshold =|Percentage of proficient students-Percentage of students above threshold] 


Percentage of proficient students whom the threshold correctly predicted = 
Number of proficient students above threshold 


Total number of proficient students 


Percentage of not proficient students whom the threshold correctly predicted = 
Number of not proficient students below threshold 


Total number of not proficient students 


Percentage of students whom the threshold correctly predicted = 

(Percentage of proficient studentsxPercentage of proficient students whom the threshold 
correctly predicted +Percentage of not proficient students<Percentage of not proficient 
students whom the threshold correctly predicted) /100 


Emerging Academic Competencies and the index of the two dimensions had similar percentages of correct 
predictions for proficient students and students who were not proficient, and these percentages were higher than 
those for Learning Engagement Competencies. Thus, the study team asked SDP to choose between using Emerging 
Academic Competencies and the index of the two dimensions, and SDP decided to use the index of the two 
dimensions. 


Because the indicator will be used to estimate the overall proportion of students who are on-track to achieve 
grade 3 reading proficiency, the study then calculated the threshold on the index of the two dimensions that came 
closest to matching the actual grade 3 reading proficiency rate. That is, the selected threshold minimized the 
absolute difference between actual and predicted proficiency. 


To select the threshold with the smallest difference between actual and predicted proficiency, students were 
ranked by their KEI index scores. The study team then attempted to identify the KEI index score at exactly the 63rd 
percentile of the distribution because the percentage of students with KEI index scores above this value (37 
percent) would match the percentage of students in the data who were actually proficient on the grade 3 PSSA in 
ELA. In practice, because students cluster at discrete KEI index scores, the study identified two KEI index scores— 
one directly above and one directly below the 63rd percentile of the distribution. The study team then selected 
the number with the smallest difference between actual and predicted proficiency (5.97) and rounded it up to 6 
to make it easier to communicate. The difference between actual and predicted proficiency for a selection of 
possible thresholds (between 1 and 10) on the KEI index score are shown in table B4. SDP can use the equation to 
create the index (equation A2) and the threshold of 6 to calculate the percentages of future cohorts of students 
with KEI scores who are on-track to be proficient in reading in grade 3. 


Different measures of accuracy (such as correctly predicting proficient students or students who were not 
proficient) could be given priority depending on the intended use of the threshold. Most predictive analyses aim 
to achieve acceptable levels of predicting proficient students or students who were not proficient, depending on 
the analyses’ goals. For example, for referring students for additional risk assessment, there may be a large benefit 
to identifying students at risk of not being proficient and a low cost to misclassifying some students as being at 
risk. This application might target more accurate identification of students who were not proficient than an 
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application that aims to provide resource-intensive supports to at risk students. Therefore, the same methodology 
described above was used to identify a cut point where at least 90 percent of students who were not proficient 
were correctly identified. Ninety percent was selected as a threshold to match existing research (see Compton, 
Fuchs, Fuchs, & Bryant, 2010; Jenkins, Hudson, & Johnson, 2007). 


Assessing the accuracy of the threshold using data that were not used to set the threshold. All four measures of 
accuracy described in box 1 were calculated using the cohort threshold of 6 for the 30 percent sample that was 
not used to set the threshold. These measures were compared to their values in the sample used to set the 
threshold. 


Research question 3: How do AlMSweb Reading scores in the spring of kindergarten through grade 3 relate to 
children’s scores on the KEI and grade 3 PSSA in ELA? 


Descriptive statistics and pairwise correlations were used to explore the relationships between spring AlMSweb 
scores from kindergarten through grade 3 and KEI and PSSA ELA scores, following the approach used in research 
question 1. Measures of accuracy (correctly predicting proficient students or students who were not proficient on 
the grade 3 PSSA in ELA) were calculated for the developer-selected AlMSweb thresholds, using the equations 
shown in box A1. 
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Appendix B. Supporting analyses 
This appendix presents supporting analyses for each research question. 


Relationships between KEI scores and grade 3 PSSA scores 

For research question 1, students with higher KEI scores on the Emerging Academic Competencies and Learning 
Engagement Competencies dimensions had higher grade 3 PSSA in ELA scores (figure 1). The numeric values for 
this figure are shown in table B1. 


Table B1. Students with higher Kindergarten Entry Inventory scores were more likely to be proficient on the 
grade 3 Pennsylvania System of School Assessment in English Language Arts 


Percentage of students who were proficient on 


Emerging Academic Learning Engagement 
KEI score Competencies Competencies 
1—1.5 13 14 
1.6—2 22 22 
2.1—2.5 35 30 
2.6—3 43 45 
3.1—3.5 63 57 
3.6—4 79 63 


Note: The table is based on the random sample of students with both KEI and grade 3 PSSA scores that was used to set the threshold and weighted to 
represent all students with KEI scores. 3,521 students had information on both KEI competencies scores and grade 3 PSSA in ELA. 
Source: Authors’ analysis based on School District of Philadelphia data 2014/15 and 2017/18 data described in appendix A. 


Regression analyses using the sample with one KEI dimension 

For the logistic regression analyses predicting PSSA proficiency (table 2), findings were restricted to students with 
both KEI dimensions. Findings were similar when all students with the relevant KEI dimension were included in 
the analyses (table B2). 


Table B2. Emerging Academic Competencies were more predictive of students’ probability of being proficient 
on the grade 3 Pennsylvania System of School Assessment than were Learning Engagement Competencies 
Relationship between KEI scores and the probability of being proficient on the grade 


3 PSSA in ELA 
Model 2: Learning 

Model 1: Emerging Engagement 
qe Me fleakeraksicelals Nor-Yol=Yanlrom Golan) eX-1n- ale) Competencies \V/KoYoK=) Be dan soya a melleatslaciiolars 
Emerging Academic Competencies 0.25*** 0.24*** 
Learning Engagement Competencies 0.25*** 0.02 
Pseudo R-squared 0.09 0.06 0.09 
Number of students 3,697 4,169 3,521 


ELA is English language arts. KEI is Kindergarten Entry Inventory. PSSA is Pennsylvania System of School Assessment. 

* Significant at p = .05; ** significant at p = .01; *** significant at p = .001. 

Note: Analysis based on the random sample of students with KEI dimension scores and grade 3 PSSA scores that was used to set the threshold. Analyses 
were weighted to represent all students with the KEI dimensions scores included in the analysis. Reported values are average marginal effects, or the 
proportional increase in the average probability of PSSA proficiency associated with a 1-point increase in a KEI score. That is, in model 1, for every 1-point 
increase in Emerging Academic Competencies, the probability of proficiency increases on average 25 percent. Pseudo R-squared is a measure of how well 
the model fits the data compared to a model with no predictors. 

Source: Authors’ analysis based on SDP 2014/15 and 2017/18 data described in appendix A. 
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Alternative weights 

For research question 2, analyses of the accuracy of the selected threshold were repeated using alternative 
weighting approaches: no weights and weights based on student characteristics to represent all kindergarteners. 
The similarity of the findings indicate that the study’s results are not overly sensitive to the weights used (table 
B3). 


Table B3. The threshold of 6 had similar rates of accuracy in samples that were and were not used to select 
the threshold using alternative weights 
PNokxo) (hn) Correct predictions for 
difference 
between actual 


and predicted Noy 
Actual Predicted proficiency Proficient proficient Number 
proficiency proficiency (percentage students students All students of 
Sample (percentage) (percentage) rexeyfales)) (percentage) (percentage) (percentage) students 


Primary analysis, weighted to represent all students with KEI scores 


Used to set the 37 37 1 53 73 66 3,521 
threshold 
Not used to set 37 37 0 58 76 69 1,508 


the threshold 


Alternative analysis, weighted to represent all kindergarteners 


Used to set the 37 36 0 53 73 66 3,521 
threshold 
Not used to set 36 37 1 58 76 69 1,508 


the threshold 


Alternative analysis, not weighted 


Used to set the 38 39 1 55 71 65 3,521 
threshold 
Not used to set 34 37 3 61 74 69 1,508 


the threshold 


KEI is Kindergarten Entry Inventory. 

Note: The primary analysis weighted the sample based on KEI scores and student characteristics. The alternative weighted analysis weighted the sample 
based on student characteristics. 

The difference between actual and predicted proficiency may not equal the difference between the actual and predicted proficiency columns in the table 
due to rounding. 

Source: Authors’ analysis based on School District of Philadelphia 2014/15 and 2017/18 data described in appendix A. 


Alternative thresholds on the index of both Kindergarten Entry Inventory dimensions 


This section presents the accuracy of alternative thresholds on the index of both KEI dimensions (table B4). The 
selected threshold for predicting the proportion of students on-track for reading proficiently by grade 3 minimizes 
the difference between actual and predicted proficiency. Another threshold that accurately predicted more of the 
students who were not proficient might be more appropriate for predicting which students need additional 
supports. For example, the threshold of 7 correctly predicted a higher percentage of students who were not 
proficient than the threshold of 6 (table B4 and figure B1). 
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Table B4. Alternative thresholds on the index of both Kindergarten Entry Inventory dimensions had different 
rates of accuracy 


Nortel Nitemeltivclactates Correct predictions for 

Threshold on the index Predicted between actual and Proficient Not proficient 

using both KE] proficiency predicted proficiency students students All students 
olfaateaksielans (percentage) (percentage points) (percentage) (percentage) (percentage) 
1.00 100 63 100 0 37 
1.25 99 62 100 2 38 
1.50 98 61 99 3 39 
1.75 99 60 99 4 39 
2.00 96 59 99 6 40 
2.25 94 57 98 8 42 
2.50 92 55 97 11 43 
2.75 89 52 96 15 45 
3.00 87 50 95 18 47 
3.25 84 47 94 22 49 
3.50 81 44 92 25 50 
3.75 78 41 90 29 52 
4.00 72 34 85 34 53 
4.25 67 30 84 42 57 
4.50 63 26 80 47 59 
4.75 58 21 76 52 61 
5.00 54 17 71 57 62 
5.25 50 13 67 61 63 
5.50 46 8 62 64 64 
5.75 40 3 57 70 65 
6.00 37 1 53 TE) (ele) 
6.25 32 5 49 78 67 
6.50 28 9 44 81 68 
6.75 23 14 39 86 68 
7.00 17 21 29 91 68 
7.25 14 24 25 93 68 
7.50 11 27 21 95 67 
7.75 8 29 16 97 67 
8.00 6 31 12 97 66 
8.25 5 32 10 98 65 
8.50 4 34 7 99 65 
8.75 3 35 5 99 64 
9.00 2 35 4 100 64 
9.25 1 36 3 100 64 
9.50 1 36 3 100 64 
9.75 1 37 2 100 63 
10.00 0 37 0 100 63 


KEI is Kindergarten Entry Inventory. 

Note: Analysis based on the random sample of students with both KEI and grade 3 PSSA scores that was used to set the threshold and weighted to represent 
all students with KEI scores. 3,521 students had information on both KEI competencies scores and grade 3 PSSA in ELA. The threshold selected for estimating 
the proportion of students on track to achieve reading proficiency is highlighted. The difference between actual and predicted proficiency may not equal 
the difference between the actual and predicted proficiency columns in the table due to rounding. 

Source: Authors’ analysis based on School District of Philadelphia 2014/15 and 2017/18 data described in appendix A. 


Figure B1. A higher threshold of 7 increased the percentage of students correctly predicted to be at risk of 
being not proficient on the grade 3 Pennsylvania System of School Assessment in English Language Arts 


Threshold 
25 
a Below threshold, predicted to not be proficient Above threshold, predicted to be proficient 
n 
| 20 
) 
ne) 
= 
=. 15 c 
ro) Incorrectly 
S predicted to 
a a 
@ 10 be proficient 
se 
c 
co) 
5 5 
2 B. Incorrectly predicted 
to be not proficient A. Correctly 
6 predicted to be proficient 
a 2 3 4 5 6 7 8 9 10 


KEI index score 


Actual proficiency status 


Not pr enielent Proficient 
Correctly predicted to be not proficient DIncorrectly predicted to be not proficient 


Dincorrectly predicted to be proficient @Wcorrectly predicted to be proficient 


ELA is English language arts. KEI is Kindergarten Entry Inventory. PSSA is Pennsylvania System of School Assessment. 

Note: Analysis based on the random sample of students with both KEI and grade 3 PSSA scores that was used to set the threshold and was weighted to 
represent all students with KEI scores. 3,521 students had information on both KEI competencies scores and grade 3 PSSA in ELA. Data are aggregated to 
smooth out the distribution, so no data are displayed at exactly 1 or 10. 

Figure reads: At this threshold, students who were actually proficient (areas A and B) were either correctly predicted to be proficient (area A) or incorrectly 
predicted to be not proficient (area B). Students who were actually not proficient (areas D and C) were either correctly predicted to be not proficient (area 
D) or incorrectly predicted to be proficient (area C). 

Source: Authors’ analysis based on School District of Philadelphia 2014/15 and 2017/18 data described in appendix A. 


Exploratory analyses of AlMSweb scores 

The final section presents additional information on the exploratory research question about the relationship 
between children’s reading skills across time (measured by the AlMSweb), their KEI scores, and their scores on 
the grade 3 PSSA in ELA. This study also present sensitivity analyses that use a consistent sample of students who 
had scores on the KEI index, spring AlMSweb in each year from K through grade 3, and PSSA. 


Students with higher Emerging Academic Competencies and Learning Engagement Competencies scores were 
more likely to be identified as on target (Tier 1) on the AlMSweb. Consistent with the patterns from the correlations 
presented in the report, students with higher KEI scores were more likely to be classified in AlMSweb Tier 1, in 
which students meet the grade-level target. For example, over 95 percent of those with the highest Emerging 
Academic Competencies scores (3.6-4) were in AlMSweb Tier 1 by the spring of kindergarten, compared with less 
than 25 percent of students with the lowest scores on Emerging Academic Competencies (1-1.5) (figure B2). By 
grade 3, about 80 percent of students in the highest range (3.6-4) were classified in Tier 1 (figure BS). Students 
with higher Learning Engagement Competencies scores were also more likely to be identified as on target (Tier 1) 
on the AlMSweb from kindergarten through grade 3 (figures B6 through B9). 
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Figure B2. Students with higher Emerging Academic Competencies scores were more likely to be in Tier 1 (“at 
target”) on the AlMSweb in spring of kindergarten 
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KEI is Kindergarten Entry Inventory. 

Note: Analysis was based on the random sample of students used to set the threshold and was restricted to the 4,930 students that had information on KEI 
Emerging Academic Competencies and kindergarten spring AlMSweb performance. The data the graph is based on was not weighted. 

Source: Authors’ analysis based on School District of Philadelphia 2014/15 data described in appendix A. 


Figure B3. Students with higher Emerging Academic Competencies scores were more likely to be in Tier 1 (“at 
target”) on the AlMSweb in spring of grade 1 
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KEI is Kindergarten Entry Inventory. 

Note: Analysis was based on the random sample of students used to set the threshold and was restricted to the 3,689 students that had information on KEI 
Emerging Academic Competencies and grade 1 spring AlIMSweb performance. The data the graph is based on was not weighted. 

Source: Authors’ analysis based on School District of Philadelphia 2014/15 and 2015/16 data described in appendix A. 
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Figure B4. Students with higher Emerging Academic Competencies scores were more likely to be in Tier 1 (“at 
target”) on the AlMSweb in spring of grade 2 
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KEI is Kindergarten Entry Inventory. 

Note: Analysis was based on the random sample of students used to set the threshold and was restricted to the 3,948 students that had information on KEI 
Emerging Academic Competencies and grade 2 spring AlMSweb performance. The data the graph is based on was not weighted. 

Source: Authors’ analysis based on School District of Philadelphia 2014/15 and 2016/17 data described in appendix A. 


Figure BS. Students with higher Emerging Academic Competencies scores were more likely to be in Tier 1 (“at 
target”) on the AlMSweb in spring of grade 3 
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KEI is Kindergarten Entry Inventory. 

Note: Analysis was based on the random sample of students used to set the threshold and was restricted to the 3,707 students that had information on KEI 
Emerging Academic Competencies and grade 3 spring AlMSweb performance. The data the graph is based on was not weighted. 

Source: Authors’ analysis based on School District of Philadelphia 2014/15 and 2017/18 data described in appendix A. 
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Figure B6. Students with higher Learning Engagement Competencies scores were more likely to be in Tier 1 
(“at target”) on the AlMSweb in spring of kindergarten 
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KEI is Kindergarten Entry Inventory. 

Note: Analysis was based on the random sample of students used to set the threshold and was restricted to the 5,446 students that had information on KEI 
Learning Engagement Competencies and kindergarten spring AlMSweb performance. The data the graph is based on was not weighted. 

Source: Authors’ analysis based on School District of Philadelphia 2014/15 data described in appendix A. 


Figure B7. Students with higher Learning Engagement Competencies scores were more likely to be in Tier 1 
(“at target”) on the AlMSweb in spring of grade 1 
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KEI is Kindergarten Entry Inventory. 

Note: Analysis was based on the random sample of students used to set the threshold and was restricted to the 4,200 students that ad information on KEI 
Learning Engagement Competencies and grade 1 spring AlMSweb performance. The data the graph is based on was not weighted. 

Source: Authors’ analysis based on School District of Philadelphia 2014/15 and 2015/16 data described in appendix A. 
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Figure B8. Students with higher Learning Engagement Competencies scores were more likely to be in Tier 1 
(“at target”) on the AlMSweb in spring of grade 2 
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KEI is Kindergarten Entry Inventory. 

Note: Analysis was based on the random sample of students used to set the threshold and was restricted to the 4,460 students that had information on KEI 
Learning Engagement Competencies and grade 2 spring AlMSweb performance (. The data the graph is based on was not weighted. 

Source: Authors’ analysis based on School District of Philadelphia 2014/15 and 2016/17 data described in appendix A. 


Figure B9. Students with higher Learning Engagement Competencies scores were more likely to be in Tier 1 
(“at target”) on the AlMSweb in spring of grade 3 
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KEI is Kindergarten Entry Inventory. 

Note: Analysis was based on the random sample of students used to set the threshold and was restricted to the 4,177 students that had information on KEI 
Learning Engagement Competencies and grade 3 spring AlMSweb performance. The data the graph is based on was not weighted. 

Source: Authors’ analysis based on School District of Philadelphia 2014/15 and 2017/18 data described in appendix A. 


Students in higher performing AlMSweb tiers in the spring of each year were more likely to be proficient on the 
grade 3 PSSA in ELA. The numeric values for figure 4, which illustrates this finding, are shown in table B5. 
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Table B5. Students in higher performing AlMSweb tiers in the spring of each year were more likely to be 
proficient on the grade 3 Pennsylvania System of School Assessment 


Proficient students Proficient students in 
tier 3 “intensive tier 2 “strategic Proficient students in 
intervention” intervention” in tier 1 “at- 
AlMSweb assessment (percentage) (percentage) target” (percentage) Number of students 
Spring kindergarten 10 24 51 4,696 
Spring grade 1 16 31 56 4,174 
Spring grade 2 2 23 66 5,027 
Spring grade 3 3 28 67 5,266 


Note: The table is based on the random sample of students used to select the threshold and the data is not weighted. 
Source: Authors’ analysis based on School District of Philadelphia data 2014—2018 data described in appendix A. 


Results for the AlMSweb analyses were similar when based on a sample of students who had scores on all of the 
assessments used in the analyses. The samples of students used in the AlMSweb analyses presented in the report 
vary across grades because they used all of the students who took each pair of assessments. To examine whether 
the differences in samples contributed to differences in findings across years, the study team conducted all of the 
analyses for a restricted sample of 2,358 students who had scores on both KEI dimensions, the AlMSweb in the 
spring of each year from kindergarten through grade 3, and the grade 3 PSSA in ELA. The patterns of correlations 
between the AlMSweb and KEI (table B6) and the AlMSweb and grade 3 PSSA (table B7) for this restricted sample 
were similar to those for the larger sample of students who took each pair of assessments. Likewise, the findings 
were similar across samples for the analyses examining the accuracy of the AlMSweb tier 1 cutoff in predicting 
grade 3 PSSA proficiency in ELA (table B8). 


Table B6. Kindergarten through grade 3 AlMSweb scores had weak to moderate correlations with Emerging 
Academic Competencies and Learning Engagement Competencies in the sample with assessments in each year 


Spring 
kindergarten Spring grade 1 Spring grade 2 Spring grade 3 
KEI (fall of kindergarten) AlMSweb AlMSweb AlMSweb AlMSweb 
Emerging Academic Competencies 0.45*** 0.37*** 0.44*** 0.38*** 
Learning Engagement Competencies 0.32*** 0.31*** 0.27*** 0.30*** 


KEI is Kindergarten Entry Inventory. 

* Significant at p = .05; ** significant at p = .01; *** significant at p = .001. 

Note: Analyses were based on the random sample of students used to set the threshold and were not weighted. 2,358 students completed the KEI, each 
spring AlMSweb assessment from kindergarten through grade 3, and the PSSA. 

Source: Authors’ analysis based on SDP 2014-2018 data described in appendix A. 


Table B7. Kindergarten through grade 3 AlMSweb scores were moderately to strongly correlated with grade 3 
Pennsylvania System of School Assessment English Language Arts scores 
Grade 3 PSSA ELA 


AlMSweb assessment Correlation 
Spring kindergarten 0.49*** 
Spring grade 1 0.53*** 
Spring grade 2 0.70*** 
Spring grade 3 0.72*** 


PSSA is Pennsylvania System of School Assessment. 

* Significant at p = .05; ** significant at p = .01; *** significant at p = .001. 

Note: Analyses were based on the random sample of students used to set the threshold and were not weighted. 2,358 students completed the KEI, each 
spring AlMSweb assessment from kindergarten through grade 3, and the PSSA. 

Source: Authors’ analysis based on School District of Philadelphia 2014—2018 data described in appendix A. 
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Table B8. The AlMSweb Tier 1 cutoff correctly predicted more than 49 percent of the students who were not 
proficient and more than 65 percent of the proficient students in English Language Arts on the grade 3 
Pennsylvania System of School Assessment 


Correct predictions for: 
Not proficient 


Proficient students students All students 
AlMSweb assessment and threshold (percentage) (percentage) (percentage) 
Spring kindergarten AlMSweb tier 1 83 49 63 
Spring grade 1 AlMSweb tier 1 65 71 68 
Spring grade 2 AlMSweb tier 1 81 73 76 
Spring grade 3 AlMSweb tier 1 76 77 77 


Note: Analyses were based on the random sample of students used to set the threshold and were not weighted. 2,358 students completed the KEI, each 
spring AlMSweb assessment from kindergarten through grade 3, and the PSSA. 
Source: Authors’ analysis based on School District of Philadelphia data 2014—2018 data described in appendix A. 
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